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1 Introduction 


Genetic Algorithms are extremely useful in narrowing down a vast search space and 
finding near-optimal solutions quickly. They can be used even when there is a lack of mathematical 
representation for the problem being solved, which is why they are used in a variety of 
reinforcement learning applications. One such application is in creating intelligent agents to play 
games autonomously. 

This helps us in our broader goals for Al as games are closed and controlled environments, 
free of real-world constraints, thus providing the perfect sandbox to gain a deep understanding of 
the behaviour of machine learning algorithms. They are quantifiable and provide us with numerical 
data (scores) to evaluate how our algorithm performs, which might be difficult or infeasible to 
gather in the real world. 

They could also be used by game developers to understand their game’s difficulty level 
and find bugs that could be exploited by players. For instance, an evolutionary algorithm was able 
to find two bugs in the game Qbert which allowed users to gain endless points (Chrabaszcz et al. 
1423). 

A neural network can be used as the “brain” of an intelligent agent, to evaluate the actions 
to be taken by the player based on its inputs. However, the parameters of neural networks must be 
trained for them to function effectively. Genetic algorithms can be used to optimally tune these 
parameters to create an efficient intelligent agent. 

However, an agent that can only function in situations similar to the training environment 
has minimal real-life applications. Machine learning algorithms are often trained on a small subset 
of all the scenarios they encounter and are expected to generalise their learning to tackle new 


situations. This behaviour is known as transfer learning. The architecture of the neural network 


and the training parameters govern how it learns and therefore can Бе varied to extend its 
capabilities. 

Flappy Bird was chosen to be the game to evaluate this research as it is simplistic, allowing 
the configuration of the game to be easily changed. It circumvents the additional complexities 
associated with various inputs, numerous possible actions by the player, and complex relationships 
between them. 

Therefore, the research question this paper seeks to answer 15 “To what extent is the ability 


of an intelligent agent to operate in unfamiliar scenarios affected by the number of hidden-layer 


neurons in its neural network and the number of generations й is trained for?” 


2 Theoretical Background 


2.1 Genetic Algorithms (GAs) 


Genetic Algorithms are heuristic search algorithms which use the principles of natural 
evolution to identify optimal solutions for the given problem. They are particularly useful in 
solving problems with a large search space, lack of mathematical representation, or having a large 
number of parameters. They yield competent solutions in short durations of time, although usually 
not the best possible solution (Wirsansky 9, 19). 

The generic pseudocode for GAs is as follows (Mallawaarachchi): 


Generate the initial population 
Compute fitness 
REPEAT 
Selection 
Cie OSSOWS ie 
Mutation 
Compute fitness 
UNTIL population has converged 


These function by maintaining a fixed-size population of individuals which are evaluated 
at every iteration. These individuals represent candidate solutions as chromosomes, which are 


arrays of genes. Each gene can be a binary digit, integer, or real number (Refer to Figure 1). 


Individual/Chromosome 


Population 


Figure 1. Population of individuals with binary-coded chromosomes. 
(Made by Candidate using Google Drawings) 

The initial population is generated by randomising the genes of each individual and is 
likely to contain inefficient solutions. Once the population is generated, a fitness score—a measure 
of the effectiveness of a candidate solution—is evaluated for each individual. 

Weighted by their fitness scores, some individuals are chosen to advance to the next 
generation with no modification, as part of a process known as selection. The genes of a few 
individuals are crossed over or swapped between chromosomes to create new individuals inserted 


into the next generation. Finally, certain chromosomes are randomly reassigned genes in a process 


named mutation to introduce new genes into the population, thus preventing a homogenous 


aggregation of individuals. The chances that mutation or crossover occur in individuals are 
controlled by the specified mutation rate and crossover rate. 

Once these three operations are conducted, the fitness of the population is computed again, 
and the process repeats until the population satisfies a predetermined stopping condition 


(Wirsansky 11-13). 


2.2 Artificial Neural Networks (ANNs) 


Input Layer Hidden Layer Output Layer 


Figure 2. Simple ANN architecture with three inputs and one output, having a single hidden layer. 


(Made by Candidate using Google Drawings) 


Artificial Neural Networks, shown in Figure 2, consist of three primary layers- the input 
layer, which takes in the data, the output layer, which forms the final output, and the intermediary 
hidden layers which can be as many as required. Each layer consists of multiple neurons, and 


each neuron is connected to every neuron in the next layer. In other words, the outputs of each 


neuron in ап л“ layer are the inputs to every neuron in ће (п + 1)* layer (Mueller and Massaron 


274-275). 
The function of each neuron is to compute a weighted sum of its inputs using the weights 
and biases assigned to the neuron for each input during training, pass it through an activation 


function which transforms the output into a specific range, and output to the next layer (Mueller 


and Massaron 272-273). 


The weights апа biases of each neuron are known as the parameters of the ANN and 
determine the magnitude to which each input influences the output. These are tuned during the 
training process through methods such as stochastic gradient descent or using genetic algorithms. 

The activation function, the number of neurons in each layer, and the network architecture 
are known as hyperparameters of the ANN, whose values are determined based on the 


application of the ANN. 


2.3 Flappy Bird 


Flappy Bird is a popular side-scroller arcade game, where players have to control a bird 
using flapping actions to fly through pipes without hitting the pipes, ground or ceiling. Therefore, 
the flaps have to be timed correctly so that the bird passes through the pipe since flapping 15 the 
only available method to control the vertical position of the bird. The game requires a sense of 
when to flap, which is acquired as players play the game. An intelligent agent will be used to do 


this artificially on a clone of the game. 


Figure 3. Screenshot of the original Flappy Bird game (Brustein). 


3 Experimental Methodology 


3.1 Tuning the parameters of an Artificial Neural Network using a Genetic 


Algorithm 


For the ANN to produce an effective output, the weights and biases of each neuron has to 
be optimally tuned. Since we do not have a differentiable fitness function, which is a requirement 
to use stochastic gradient descent, we use a GA to tune the parameters instead (Kwiatkowski). 

Each gene would represent either the weight or bias of a neuron and a chromosome would 
represent all the weights and biases of the network. The GA would be used to determine the optimal 
weights and biases for the ANN, and the ANN would be used to play the game once trained. The 
outcome of the game played using each individual”s genes will enable the GA to evaluate its fitness 
score and judge its effectiveness relative to other individuals. 

The GA is enumerated through numerous generations until the ANN reaches a 
predetermined condition of accuracy and consequently, the ANN is considered trained and can be 


used as an intelligent agent to play the game. 


3.2 Architecture of the Artificial Neural Network controlling Ше bird 


Vertical 
position of 
the bird 


bottom pipe 


Distance 
from next 
top pipe 


Number of neurons here will vary. 


Input Layer Hidden Layer Output Layer 


Figure 4. Architecture of the ANN used to control the actions of the bird (Candidate). 


The ANN controlling the actions of the bird will take five inputs as shown in Figure 4, 


which correspond to the distances labelled in Figure 5. 


Flappy Bird 


Figure 5. Annotated screenshot representing the inputs of the Artificial Neural Network (Candidate). 


The sole output of the ANN will be a floating-point value between 0 and 1. Ifthe output > 
0.5, the bird will Пар, and if output <= 0.5, no action will be taken. To prevent extremely large 
outputs, we use the sigmoid activation function, which takes in the output of the neuron and 
manipulates it to satisfy the constraint 0 < output < 1. 


Overfitting is when a machine learning model “memorises” how to react to the training 


data and is unable to replicate its ability during testing despite performing exceptionally with the 


training data because its output is completely based on its memorization (Mueller and Massaron 
161). The network will only have one hidden layer to avoid overfitting since the scenario is not 
complex enough to justify the use of multiple hidden layers, which would also increase the time 


and resources required to train and execute the network. 


3.3 Parameters of Ше Genetic Algorithm 


The game score, which increases when an agent successfully passes through a pipe pair, 
will be weighted the highest in the fitness equation to reward the agent as maximising this value 
15 Our primary goal. However, the distance travelled by the bird will also be factored into the 
equation, although weighted lesser, to differentiate between birds with the same score but reaching 
a higher distance before colliding with an object. Hitting a pipe, the ground, or the ceiling result 
in an immediate termination of a game. Hence, we impose a penalty on these to reduce the fitness 
value to disincentivize the agent from repeating actions that lead to this state. 

Therefore, the fitness of the bird is given by the equation: 

Fitness = (3 x Score) + (0.1 X Distance travelled by bird) — Penalties 

We will be having a constant population size of 90 individuals for every generation, to 
avoid lag on the system as the individuals will be evaluated concurrently. Elitism will be 
implemented to preserve the best individuals in the population, although only one individual will 
remain unchanged across generations to allow for greater variance within the population. 

The selection method being used will be Rank-Based Selection, where individuals are 
ordered by their fitness values, and the probabilities of each individual being selected are 
calculated based on their rank, instead of their raw fitness values. This selection method will ensure 
that some individuals with abnormally large fitness values will not overshadow the other 
individuals and dominate the population, instead providing a substantial opportunity for all 
individuals to be selected. 

A normally distributed, also known as Gaussian mutation, will be used since the genes 
representing the weights and biases of the neurons are real numbers and Gaussian mutation is 


compatible with real-encoded genes. It generates a pseudorandom number following a normal 
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distribution (Wirsansky 46). An advantage of this method is that the generated genes are close to 
the original gene, preventing the pre-existing gene from being completely eliminated from the 
population. The mutation rate is set to a relatively high value here, 0.4, to allow new genes to 


enter the population, thus preventing convergence at less-efficient local maxima. 


Figure 6. Visual representation of Blend Crossover (Wirsansky 43) 


Blend Crossover will be used as the crossover method. Using this method, the resultant 
gene will be randomly generated from an interval derived from its parents” genes. The alpha (a) 
value 15 used to control how wide this interval is. The higher the alpha value, the wider the interval. 


Figure б represents this, where рі and p2 are the genes of the two parents. The green region 


represents the interval which the gene will be randomly chosen from. We will be using а - 0.5 


to allow for greater variance in the population (Wirsansky 42). The crossover rate will also be set 


at 0.4 for the same reason, as it allows for more variance across generations. 


3.4 Experimental Design 


3 Trials will be conducted and averaged for each experiment with different random seeds 
(namely 14, 4, and 26) to prevent the state of the randomly-generated initial population from 
skewing the results. The same 3 random seeds will be used for all the experiments for uniformity, 
since repeating the experiment specifying the same seed will yield identical results. A cap of score 
100 will be kept for each generation as without it, some excellent candidates will fly almost 


infinitely, blocking the program from advancing to the next generation. 


3.5 Experimental Procedure 


3.5.1 Experimental Setup 


The algorithm will be trained on the original version of the game, and then tested on both 
the original version and altered version where the height of the gaps between the pipes is reduced 
from 120 units which 1t was trained on to 110 and 105 units, to examine how the intelligent agent 
functions. The number of generations the GA is allowed to run for and the number of hidden- 
layer neurons will be varied separately to observe their impact on its performance in the altered 


version of the game. 


3.5.2 Experiment 1: Varying the number of neurons in the hidden layer 


The neural network saved at generation 200 of each trial will be used to compare. The 
neural networks used will be from the same generation for all the trials. The 200% generation was 
selected to ensure the individual being evaluated is already trained so the results are not skewed 


by a partially-trained agent. 


3.5.3 Experiment 2: Varying Ше number ої generations trained 


Although the number of hidden-layer neurons is kept constant within an experiment, we 
will conduct three distinct experiments with different numbers of hidden-layer neurons—one low 
(2), one medium (15), and one high (30)—so that we can observe trends, if any, between the 
different neural network architectures in the effect of varying the number of generations trained. 
We start at the 150% generation as all different hidden neuron configurations chosen have 


converged by this point. 


4 Hypothesis 


From my theoretical knowledge, І hypothesize that а higher number of hidden-layer 
neurons should result in the agent gaining а deeper learning of the game. Therefore, the agent 15 
more likely to perform better when placed in different game settings. It will be able to model more 
complex relationships and may be more accurate in its outputs. However, the performance of the 
agent in both the original and the modified game settings may be adversely affected 1f the number 
of hidden-layer neurons is too high since overfitting may occur. 

Similarly, training for more generations should provide more individuals the opportunity 
to gain optimal genes through mutation and crossover, and therefore make them more likely to be 
high-performing individuals. Hence, these agents should perform better in the modified game 
settings. Due to the implementation of elitism, it is unlikely that optimal genes will inadvertently 


be eliminated in future generations. 


5 Experimental Results and Analysis 


5.1 Experiment 1: The effect of varying the number of hidden-layer neurons 


5.1.1 Experimental Data 


Max Fitness, Median Fitness and Mean Fitness for 1 Hidden 
Layer Neuron (Seed: 14) 


== Max Fitness == Median Fitness == Mean Fitness 


60 


Generation 


Figure 7. Variation of the fitness values of the population over generations (1 hidden layer neuron)! 


In Figure 7, the max fitness shows the fitness of the best-performing individual in the 
population, and the median and mean fitness can be used to identify what proportion of the 
population is trained. Initially, the fitness values are clustered around zero. As mutation occurs, 
new genes are introduced into the population through a few individuals, shown by the observation 


that only the max fitness rises. This value fluctuates across generations, eventually reaches high 


| Refer to Appendix 9.3.2 (page 52) 


levels, signifying Фе tuning of the weights of the neural networks until it converges to a fitness 
value of approximately 400 which corresponds to a score of 100, which we limited the training 
until. Since we use elitism in the genetic algorithm to retain the best-performing bird without 
mutation, these genes are not eliminated and hence the max fitness curve remains constant from 
now. 

The mean and median curves show slow but steady increases, which illustrates the rest of 
the population being optimised through random mutation and crossover with the top-performing 
individuals. The median fitness reaching 400 represents the top 50% of the population being 
trained. However, this curve fluctuates slightly in later generations as mutation and crossover can 
cause the genes of optimal individuals to be overwritten, thus explaining the need to preserve 


optimal genes through elitism. 


Max Fitness, Median Fitness and Mean Fitness for 15 Hidden Layer 
Neurons (Seed: 14) 


== Max Fitness == Median Fitness == Mean Fitness 


100 


Generation 


Figure 8. Variation of the fitness values of the population over generations (15 hidden-layer neurons)? 


2 Refer to Appendix 9.3.3 (page 57) 


Comparing Figures 7 and $, it is immediately noticeable that it takes more generations for 
the max fitness to converge with 15 hidden-layer neurons (114), compared to with 1 hidden layer 
neuron (39). The max fitness in Figure 8 fluctuates considerably, with a generally increasing trend, 
until a sudden fluctuation causes it to converge. This could be because of a new set of genes 
introduced in the population through mutation, or through the crossover of two individuals creating 
an optimal individual. Similar to Figure 7, the mean and median fitnesses slowly approach the 


max fitness curve as the optimal genes spread across the population through crossover. 


Average number of generations required to train vs. number of hidden layer 
neurons 


== Average number of generations required to train Trendline: R? = 0.793 
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Number of hidden layer neurons 


Figure 9. Average number of generations required to train the ANN against the number of hidden-layer 


neurons.? 


Figure 9 represents the trend of how the average number of generations required to train 


the neural network is correlated with the number of hidden-layer neurons. The y-axis values are 


3 Refer to Appendix 9.3.1 (page 51) 


obtained by observing when the max fitness curve converges to 400 for each number ої hidden- 
layer neurons. There is evidently a broader trend where neural networks with more hidden-layer 


neurons take longer to train since there are more genes that need to be optimised, as shown by the 


trendline in Figure 9. 


Average score attained in modified game settings vs. Number of 
hidden layer neurons (Pipe Height 110) 


41 


Score attained in modified game settings 


20 30 


Number of hidden layer neurons 


Figure 10. The correlation between the number of neurons in the hidden layer and the average score the 


agent attained in the modified game with a smaller pipe height of 110 units* 


In Figure 10, we see that the score initially begins high, but suddenly drops апа gradually 
increases again as the number of hidden-layer neurons increases, reaching a peak at 25 hidden- 


layer neurons, consequently falling rapidly again. This confirms our hypothesis that the score will 


increase as the number of hidden-layer neurons increases since the agent achieves deeper learning, 


but the scores remain low beyond 30 neurons. 


4 Refer to Appendix 9.3.4 (page 65) 


Average score attained in modified game settings vs. Number of 
hidden layer neurons (Pipe Height 105) 
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Score attained in modified game settings 


20 30 


Number of hidden layer neurons 


Figure 11. The correlation between the number of neurons in the hidden layer and the average score the 


agent attained in the modified game with a smaller pipe height of 105 units? 


Figure 11 depicts the performance of the bird with the pipe height reduced further than in 
Figure 10. However, in the graph, we see no clear trend in scores. Therefore, the average scores 
are uniformly lower, reaching a maximum score of only 7, while in Figure 10, they reached a 


maximum average score of 41. 


5.1.2 Result Analysis 


It is clear from the collected data that the number of hidden-layer neurons corresponds to 


the number of generations required to train the network. This is because the number of possible 


combinations of neuron weights significantly rises with the number of neurons, thus complicating 


the process of finding an optimal configuration. 


5 Refer to Appendix 9.3.5 (page 65) 


The initially high score with pipe height 110 occurs because the relationship between the 
inputs and the output is simple, allowing networks with lesser neurons to function effectively. 
However, this advantage disappears when the number of neurons increases. Following this, the 
upward trend validates our hypothesis, but the scores fall beyond 30 neurons as a result of 
overfitting—these networks memorised the original environment during training and therefore 
were unable to perform when the game settings are modified. 

However, there is no clear trend in scores with pipe height 105 since the testing 
environment is significantly modified from the training environment and thus is too difficult for 
the agent to operate in. We observe that the greater the extent to which the environment is 


modified, the lower the effectiveness of the agent is. 


5.1.3 Evaluation 


The data from at least two of the three trials were often similar for each case, thus 


enabling us to validate the reliability of the data. The data from pipe height 105 enabled us to 


realise the limits of transfer learning while the data from pipe height 110 illustrated the optimal 


range of neuron configurations to utilise. 


5.2 Experiment 2: The effect ої varying the number of generations trained 


5.2.1 Experimental Data 


Score attained in modified game settings vs. Number of 
generations trained for 


== 2 Hidden Neurons == 15 Hidden Neurons == 30 Hidden Neurons 


Score attained in modified game 


Number of generations trained for 


Figure 12. Graph of the scores attained in the modified game settings when the number of generations 15 


varied from 150 to 200, with three different hidden neuron configurations® 


In Figure 12, we see from the linear trendlines plotted that there 15 a universal tendency for 
the score to increase as we increase the number of generations trained, although the extent to which 
1t varies differs. However, there are fluctuations due to large amounts of randomness propagating, 
despite the graph being plotted from an average of experiments using three different seeds to 


minimise this. 


9 Refer to Appendices 9.3.6, 9.3.7, 9.3.8 (pages 66, 68, and 71 respectively) 


Although all three curves follow an increasing trend, there is a greater degree of fluctuation 
when 30 hidden-layer neurons are used, and the peaks are the tallest here. There are moderate 


fluctuations with 2 hidden-layer neurons and minimal fluctuations with 15 hidden-layer neurons. 


5.2.2 Result Analysis 


The score increases with the number of generations due to two reasons. Firstly, the optimal 
neurons crossover with other individuals to potentially form better individuals. Secondly, there is 
a higher chance that a mutation would provide optimal genes to a random individual to overtake 
the previous best individual. 

The greater degree of fluctuation with 30 hidden-layer neurons suggests that there 15 greater 
instability in their genes, which is expected as there are more possible combinations of weights 
and biases. There is a moderate degree of fluctuation with 2 hidden-layer neurons due to the greater 
relative importance of each weight, once again causing instability. The network with 15 hidden- 


layer neurons shows minimal fluctuations due to striking a balance between these two behaviours. 


5.2.3 Evaluation 


Conducting the experiment with three different neuron configurations and attaining 
similar results indicating a direct correlation between the number of generations and the score 
validates our hypothesis for most ANNs. It also allows us to realise which situations the trade-off 


in training time is worth the gain in score resulting from better-optimised ANNs. 


6 Limitations and Potential Improvements 


In this investigation, only three trials were used and their results averaged, which meant 
that extremely-large or extremely-small anomalies skewed the resultant data significantly. To 
counter this, a higher number of trials could be conducted and their median used instead of their 
mean to curb the effect of outliers and ensure a more accurate result. 

While the neural network saved at the 200' generation from each trial was compared in 
this investigation, this provides a slight advantage to networks with lesser hidden-layer neurons as 
their scores converge to 100 earlier, thus allowing them more opportunities to enhance their 
performance. Furthermore, the score cap of 100 for each generation could be increased since the 
agent attains this by chance rather than by skill in some trials. 

The mutation rate can be set lower to prevent useful genes from inadvertently being 
eliminated from the population. This, including increasing the number of individuals retained 
through elitism, would reduce the rapid fluctuations of the fitness in the graph, although the 


increase in score would take longer. 


7 Conclusion 


This investigation successfully explored how varying the number of hidden-layer neurons 
and the number of generations trained affects the ability of an agent to adapt to a modified 
environment. 

Our hypothesis that the effectiveness of the agent in the modified environment improves 
as the number of hidden-layer neurons increases was true to some extent. Although this trend was 
upheld for small numbers, upon exceeding a certain limit, presumably determined by the 
complexity of the relation between the inputs and outputs, the effectiveness dropped due to 
overfitting. Furthermore, we observed that when the environment is modified to a greater extent, 
there 15 no clear trend since the effectiveness 15 low for all hidden-layer configurations. 

However, we also observed that as the number of hidden-layer neurons increases, the 
training time also increases multifold. This limitation could offset any potential gain provided by 
the marginal increase in effectiveness for some use cases. 

The second experiment validated our hypothesis that as the number of generations we train 
the neural network increases, its effectiveness in the modified environment improves. Yet, we 
noticed that the extent to which this matters is non-constant and varies based on the neuron 
configuration. 

However, one significant limitation of this investigation was that only one game was used 
and the environment was varied only by switching the height of the pipe gaps. In the case of Flappy 
Bird, even one hidden-layer neuron is sufficient for the neural network to operate as the 
relationship between the inputs and the output is simple, which means that increasing the number 
of hidden-layer neurons makes it more complex and could lead to worse performance as well. 


However, this may not be the case for games with greater complexity, and therefore further 
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experimentation is required before this finding can be generalised. Using more methods to vary 
the environment would also allow us to be more certain of our results. 

In this investigation, our inputs were limited to 5, and all the hidden-layer neurons were 
placed in one layer. Future exploration into how the quantity and nature of the inputs to the neural 
network or the number of hidden layers in the neural network affect its ability to perform in novel 
environments can prove beneficial. 

Therefore, the research question “To what extent is the ability of an intelligent agent to 
operate in unfamiliar scenarios affected by the number of hidden-layer neurons in its neural 


network and the number of generations й is trained for” can be concluded with this 


investigation, which exhibits that the hidden-layer neurons and the number of generations do 


affect the effectiveness of a neural network in new environments, although the degree of 


correlation varies in certain cases. 


8 Bibliography 


Works Cited 

Brustein, Joshua. “The Mysteries of Apps: Flappy Bird Shows That Dumb Luck Matters.” 
Bloomberg.com, 7 February 2014, https://www .bloomberg.com/news/articles/2014-02- 
07/the-mysteries-of-apps-flappy-bird-shows-that-dumb-luck-matters. Accessed 14 
September 2022. 

Chrabaszcz, Patryk, et al. “Back to Basics: Benchmarking Canonical Evolution Strategies for 
Playing Atari.” 2018, рр. 1419-1426, https: //www.ijcai.org/proceedings/2018/0197.pdf. 
Accessed 20 June 2022. 

Kwiatkowski, Robert. “Gradient Descent Algorithm — a deep dive.” Towards Data Science, 22 
May 2021, https://towardsdatascience.com/gradient-descent-algorithm-a-deep-dive- 
cf04e8115f21. Accessed 18 January 2023. 

Mallawaarachchi, Vijini. “Introduction to Genetic Algorithms — Including Example Code.” 
Towards Data Science, 2017, https://towardsdatascience.com/introduction-to-genetic- 
algorithms-including-example-code-e396e98d8bf3. Accessed 7 June 2022. 

Mueller, John Paul, and Luca Massaron. Machine Learning For Dummies. Wiley, 2021. 
Accessed 12 June 2022. 

Wirsansky, Eyal. Hands-on Genetic Algorithms with Python: Applying Genetic Algorithms to 
Solve Real-world Deep Learning and Artificial Intelligence Problems. Packt Publishing 
Limited, 2020, https://www .packtpub.com/product/hands-on-genetic-algorithms-with- 


python/9781838557744. Accessed 26 May 2022. 


9 Аррепаїх 
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Figure 13. Screenshot taken while conducting the experiment (Candidate). 


9.2 Codebase used for experiment 


(Adapted from https: //github.com/crO0nkz/Sloppy-Block under the GPL3 licence by modifying 


the genetic algorithm and neural network, modifying the game code, and including logging of 


readings) 
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blockPic = pygame.image.load 


upperPipePic pygame.image.load ( 


lowerPipePic pygame.image. load ( 


backgroundPic = pygame.image. load ( 


cloudPic = pygame.image.load ( 


pygame.display.set icon(blockPic) 


player = 
multiPlayer = (|| 
pipes 


elouds 


nning = 
pygame.font.SysFont ( 
ittlefont = pygame.font.SysFont ( 


generation = 


birdsToBreed = [] 
highscore = 0 
highgen = 0 
allTimeBestBird = 
maxscore = 
singlePlayer = 
globalFitness = 


respawn = 


player, running, score, multiPlayer 


singlePlayer 


respawn 


len (pipes) > 
pipes.pop(0) 


ae 9): ()) 


init pipe (w=WIDTH + WIDTH / 2) 


len(clouds) > 
clouds.pop (0) 
init cloud (w=0) 


init cloud (w=WIDTH / 2) 


init cloud (w=WIDTH) 


running = 


EVAL MODE GEN MODE: 


agent = bird.Bird(HEIGHT, num=len(evalHiddenWeights) ) 


agent.setWeights (evallnputWeights, evalHiddenWeights) 


multiPlayer = [agent] 


singlePlayer = bird.Bird (HEIGHT, num=num) 
len (birdsToBreed) == 


range (BIRDS): 


multiPlayer.append (bird.Bird (HEIGHT, num=num) ) 


multiPlayer 


ELITISM: 


_ = ш Јел зао | Mela Gian, arena) 
бен Мені ке рана (birds ToBreca [10]. реле 5и 


birdsToBreed[0].hiddenWeights) 


multiPlayer.append(_ 


sortedPlayers = sorted (birdsToBreed, key=lambda player: 


player.fitness) 


totRanks = len(sortedPlayers) * (len(sortedPlayers) + 2 


for rank, player іп enumerate (sortedPlayers, start=1): 


probability = rank / totRanks 


wheel [ ] .append (player) 
wheel [ ] .append (probability) 
selecues players = ma. каш ыш оше шее 
len (multiPlayer), ромпее! | 1) 


nen pleyers = [| 


woe plays alía selecues! players: 


if random.random() > CROSSOVER RATE: 


player = bird.Bird(HEIGHT, player, 


np.random.choice(selected players), num=num) 


else: 


player = bird.Bird(HEIGHT, player, num=num) 


if random.random() > MUTATION RATE: 


player.mutate() 


new players.append (player) 


multiPlayer.extend (new players) 


def init pipe (w=WIDTH): 


dist = 
for pipe in pipes: 


dist = pipe.x 


return pipes.append (Pipe (w, HEIGHT, dist, pipeheight=PIPEHEIGHT)) 


def init cloud (w=WIDTH): 


return clouds.append (cloud.Cloud (м, Hl 


def draw (window): 


if HIGHDETAILS: 


window.blit (backgroundPic, (0, )) 


a СИ оше 


window.blit (cloudPic 


pipe pipes: 


Пао Том іле (energie, (оное, Dips asa y - № 


window.blit (lowerPipePic, (pipe.x, pipe.lower y)) 


pygame.draw.rect (window 
ріре pipes: 
pygame.draw. rect (window 


pipe.upper y)) 


pygame.draw. rect (window 255, Ore 10 Piper оте у 


drewBird = 
рјауек multiPlayer: 
player.alive: 


HIGHDETAILS: 


topleft = (BLOCKSIZI player.y) 


rot = player.velocity * -5 
zoe < = 05 


rot = -90 


аас lloc = ЭМИ erens Torm coreo (ПОШ (ехе! ЫЕ, вое) 


new ect = moteles! lolocl: GEE вое | 


COMES NOS PS @үг mear (ООО @зї с езе 


О ОТЕЛЕ Е о азе ют ос, me CSCE оре) 


if birdView: 
pygame.draw.line (window, (0, „ О), 


( + BIOCKSIZE 7 2, 


ріауег.у + BLOCKSIZI 


(BLOCKS тула + player.distanceX, 


player.y + player.distanceTop) ) 


pygame.draw.line(window, (0, 0, ), 


( + ВОС / 2, 


ріауег.у + BLOCKSIZI 


(BLOCKSIZE / + player.distanceX, 


player.y + player.distanceBot) ) 


pygame.draw.line (window, ( 


(20 + BLOCKSIZE / 2, 


+ BLOCKSIZE / 2, 


), 


ріауег.у + BLOCKSIZI 


player.y + BLOCKSIZE / 


player.velocity)) 
pygame.draw.line (window, ( 


(20 че патио бен / 2, 


+ BLOCKSIZE 


ріауег.у + BLOCKSIZI 


player.velocity)) 


elif (not HIGHDETAILS) and (not drewBird): 


pygame.draw.rect (window, (0, 


(20, player.y, BLOCKSIZE, 


drewBird = True 


), 


ріауег.у + BLOCKSIZI 


BLOCKSIZI 


draw text ( 


n 


textColor 


HIGH 


textCol 


alive: 


text 


window. 


text 


window. 


text 


window. 


text 


window. 


text 


alive 


oAlive= 


(0 


DETAILS: 


Que 


(0 


font.render ( 


blit (text 


littlefont 


БЛ (Ж ех 


littlefont. 


БИЛЕ (Б ех 


ШЕЕ ЕШ ГЕ 


blit (text 


ЕЕ РОЛЕ 


ОЕ (15 ӨБ 


window. 


text 


ПЕЕ ЕЛ Ет ЕЕ 


blit (text 


Liec leront, 


(ОЛА 


.render ( 


highscore, fitness= maxGen= 


ЕНЕ) = ЛЕ 


. format (score) textColor) 


(WIDTH / - text.get width() // 


0)) 

. format (round (fitness, 2)) 
textColor) 

(WIDTH = Ese ger музея), ©) )) 

render ( . format (gen) 

textColor) 

(ШИШЕ — еее Gel Ghea) ere. ger mexgiae (0) ))) 

render ( . format (round (maxscore 

textColor) 

(WIDTH - text.get width() Ere GEE телу) 

render ( . format (maxGen) 

textColor) 

(UCD = exe бе (Көніл (0) Were селе mece (0) 

render ( . format (noAlive) 


еее) 


(ПЕСНЕ = ех eje коем (0) 


Were Сеп яние (0) 


render ( 


. format (num) textColor) 


window.blit (tex (WIDTH - text.get width() pere eee тес аа) 


text = littlefont.render ( : . format (SEED) tex 


window.blit (text (WTD = tert Ger л ЕД перс qe mel © 


text = littlefont.render ( . format (MUTATIO 
textColor) 


window.blit (text (WIDTH - text.get width() text. GEE шке си ()) = 


text = littlefont.render ( · format (CROSSOVER RAT 
textColor) 


window.blit (text (MALO TE = exe gee коем (0) were „See mece (у $ 


text = font.render ( 


window.blit (text (WIDTH / 2 - text.get width() 


HEIGHT / : text.get height () 


text = font.render ( . format (score) 


window.blit (text (WIDTH / 2 - text.get width() // : 01) 9) 


text = littlefont.render ( . format (round (highscore, 2)) 
(128 0 0)) 


window.blit (text (WIDTH - text.get width () vere ger шел бйз = 2)) 


EVAL MODE: 


filename = 


open (filename ЕВ 
reader = list (csv.reader (f, delimiter= ) ) 
reader [1] [2] 


row reader[1:]: 


SEENE 


evalInputWeights eval (row[5] .replace ( 


evalHiddenWeights = eval (гом [6] .геріасе ( 


СЕМ МОРЕ: 


filename = input ( 


vell зао узоре = || 


minerede im E сому = 
open (filename 
reader = list (csv.reader (Е, delimiter= )) 
кеадек [1] [2] 
row reader [1:]: 


hundreds in a row >= 


generation = int (row[3]) 
evallnputWeights = eval(row[5].replace ( 


evalHiddenWeights = eval(row[6].replace ( 


eval individuals.append( (generation, evalInputWeights 
evalHiddenWeights) ) 


str(row[4]) == 


Imbiacliascs sa ашк к= 


ao (ajo (( 


СЕМ МОРЕ: 


eval individuals 


individual eval individuals: 
individual: 


generation, evallnputWeights, evalHiddenWeights = individual 


draw (window) 


currentfitness = 0.0 


event pygame.event.get () : 
even AOS == yes, (OIE 


sys.exit () 


running: 


© ЛЕШЕ Су 


воно dert) 


e 6 = Д0) о 


male, Cowie) 


clouds.pop(0) 


pipes: 
15,395 < 50: 


рірез.рор(0) 


mal, оваке (()) 
зсоке += 1 
рјауек multiPlayer: 
player.alive: 


player.fitness += 


p.move left () 


noAlive = 


р = pipes[0] 


player multiPlayer: 


player.alive: 


(EVAL MOD 


score >= SCOR 


player.alive 


player.velocity += 1 


player.handleCollision (HEIGHT, BLOCKSIZI 


player.alive: 


player.y += player.velocity 


noAlive += 


Б ламен ресовезвевиатт(Е- повете ЖО ШЕ е Шу ме. >) 


currentfitness = player.fitness 


globalFitness = player.fitness 


player.thinkIfJump (): 


player.velocity = VELOCITYGAIN 


noAlive == 


running = 


range (len (multiPlayer)): 


player = multiPlayer[i] 


(player.fitness > highscore) 


player.bestReported): 


player.bestReported = 


draw text (alive= fitness=currentfitness 


gen=generation 


maxGen=highgen, noAlive=noAlive 


ES 


оссте=зсоте 


highscore=maxscore) 


EVAL MODE СЕМ МОРЕ: 
cime == оно (есіне (1) ) 


лыг velwes аел ее player multiPlayer] 


open ( run time 


writer = csv.writer(f, delimiter= 


йаш гиена гн б (lesa, Sal generation, РІРІ 


Score, round (тах (fitness values) 2) 


round (np.median (fitness values) 


round (np.mean (fitness values) 2.11) 


ог (maxscore > 0) ог (globalFitness > 


birdsToBreed = [] 

for ІП in rangel2): 
bestBird = - 
bestFitness = - 


for i in range (len (multiPlayer)): 


player = multiPlayer[i] 
if player.fitness > bestFitness: 
bestFitness = player.fitness 
bestBird = 
if bestFitness >= highscore: 
highscore = bestFitness 


) апа (bestFitness >= highscore): 


allTimeBestBird = multiPlayer[bestBird] 

bestInputWeights = copy.deepcopy ( 
multiPlayer[bestBird] .inputWeights) 

bestHiddenWeights = copy.deepcopy ( 
multiPlayer[bestBird] .hiddenWeights) 

highscore = bestFitness 

highgen = generation 


шазхзсове = соге 


for ) in range (len (multiPlayer)): 


birdsToBreed.append (copy.deepcopy (multiPlayer[j])) 


eine = лише (imah) 


ес valies = player. кыа ебе рјауек multiPlayer] 
open ( пто пеў fa 


writer = csv.writer (Е, delimiter= 


writer.writerow([time , SEED, num, generation, score 


moss (manes soles) np.median(fitness values) 


Пре (шев уа е5) 
ореп ( оо те ) во 


writer = csv.writer (Е, delimiter= 


МОЕ  Weleeszow(esme y SE generation, score 


repr (bestInputWeights), repr (bestHiddenWeigh 


ува ета тат Е слао (eme , SHED), mun, cenezsi Loa, SCOTS 


repr (multiPlayer[0].inputWeights) repr (multiPlayer[0].hiddenWeights) ]) 


generationi CIN ПОЉ ЕШШ: 


Peiner ) 


generation += 


чета и НЕ (0) 


pygame.display.update () 


POS о Е (ЕЕ ЕС С 


питру пр 


капдот 


height, parentl= parent2= 


.bestReported = 


у = heigat / 
.velocity = 
.distanceBot 
.distanceTop 
.distanceX = 0 
.distanceGround = 


.distanceCeil = 


parentl == parent2 == 
.inputWeights = np.random.normal (0 
-hiddenWeights = np.random.normal (0 


parentl != parent2 == 


.inputWeights = parentl.inputWeights 


-hiddenWeights = parentl.hiddenWeights 


.inputWeights = np.random.normal (0 


-hiddenWeights = np.random.normal (0 


. Crossover (ракеп 1, parent2) 


pipeUpperY, pipeLowerY, pipeDistance): 


tanceTop = pipeUpperY 


tanceBot = pipeLowerY 


tanceX = pipeDistance 


«fitness += 0.01 


EIGHT, BLOCKSIZ 


((pipe.x >= 20) (pipe.x <= 20 + BLOCKSIZI ((pipe.x + 20 >= 


(pipers в 20 <= 20I BLOCKS An) i: 


.alive (( sW <= pipa- uasez y) sy ар SHOCKS ILAI 


>= pipe lower у) ) s 
Шла = 


.fitness -= 1 


-Velocity > HEN GENE BLOCKSIZE: 


Y = ИКО = BLOCK 


.alive = 
.fitness -= 1 


F VELOC < 15 


.alive = 


.fitness 


- Y .distanceBot .distanceTop .distanceX 
.velocity] 


аке laysar ma = паре dee .inputWeights) 


Ой ауе ovit = .sigmoid (hidden layer in) 


стерео leyer su пр. Clore (шега а е layer Ole .hiddenWeights) 


prediction = .sigmoid (output layer in) 


prediction > 


inputWeights, hiddenWeights) : 


.inputWeights = inputWeights 


-hiddenWeights = hiddenWeights 


male, female, alpha=0.5): 


range (len(self.inputWeights) ): 


range (len ( .inputWeights[i])): 


renge = (female.inputWeights[il[3] - 
male.inputWeights[i][3]) 

lower = min(female.inputWeights[i][3], 
г: зрацима a) = alas * ва _ 

upper = (female.inputWeights[i] [j], 
Tele. тотар ае ІЗІ ІШІ) = aljama = range _ 


.inputWeights[i][j] = lower + random.random() * (upper - 


Реле йо Sha ( 1f.hiddenWeights)): 
шөлге 5) оба (self .hiddenWeights[i])): 


range = (female.hiddenWeights[i][j] - 


male.hiddenWeights[i] [j]) 


lower = (female. hiddenWeights[i][j], 


mele пен еее [ai] IS) = ге 7 zange | 


upper = ах (female.hiddenWeights[i][jl, 


male.hiddenWeights[i][3]) + alpha * range 


-hiddenWeights[i][j] = lower + random.random() * (upper - 


def mutate ( 


Реле йо skin лаје self.inputWeights)): 
вое 5) Jm ae ге ( 1 (s€ .inputWeights[i])): 
.inputWeights[i][j] = 
F.getMutatedGene ( (f.inputWeights [1] (391) 


Bone 40 ай і (se -hiddenWeights) ): 


range (len ( -hiddenWeights[i])): 
.hiddenWeights[i] [j] = 


.getMutatedGene ( -hiddenWeights [i] [j]) 


Ме ата о стао random eme |00, 23) = 0.005 


mutatedWeight random.gauss (weight, learning rate) 


mutatedWeight 


gereen ие ПЕ пасе helgat, Clisiceimee Тото Це 


pipeheight): 


= ancom- licencia (10, оао keleme — 


.Upper y пеге ћоко Кош Секен M (0, еее las Lola = 


-.lower y Le Әле y + (Әз ое кезі ae 


.x = distanceToOldPipe / П Белае а ое се зао (елите газоне (0), 


Cle move lerre 


cloud.py 


import random 


class Grouch: 


WIDTH + + random.ra 


Чу random.randint(0, 1 (НІ 
[НЕ ОКА НЯ Б = 


.moveTick = 


6 ALC UMIEN 


.moveTick += 


9.3 Data Tables 


9.3.1 Hidden-Layer Neurons vs Training Time 


1 29 26 25 37 


2 49 44 45 59 


38 26 46 41 


86 77 68 114 


67 67 81 53 


66 46 53 


96 79 66 


9.3.2 Мах, Median and Mean Fitness for 1 Hidden Layer Neuron (Seed: 14) 


1 0.52 


-0.79 -0.71 
0.51 -0.79 -0.67 
0.52 -0.78 -0.62 
0.28 -0.78 -0.65 
0.52 -0.78 -0.40 
0.54 -0.78 -0.45 
4.12 -0.29 -0.13 

0.51 0.52 

0.54 0.51 

0.40 0.24 

0.53 -0.04 

0.36 0.14 

0.50 

0.51 

0.51 

0.50 

0.53 

-0.79 

0.57 

0.54 


0.39 


9.3.3 Max, Median and Mean Fitness for 15 Hidden Layer Neurons (Seed: 14) 


1 0.52 


-0.78 -0.65 
0.51 -0.78 -0.64 
4.73 -0.78 -0.54 
4.82 -0.78 -0.50 
0.51 -0.78 -0.52 
-0.78 -0.38 
-0.78 -0.28 
-0.78 -0.15 
-0.78 -0.17 
-0.78 -0.52 
-0.78 -0.08 
-0.78 -0.46 
-0.78 -0.02 
-0.78 -0.03 
-0.78 -0.37 
-0.78 -0.18 
-0.78 -0.24 


-0.78 -0.01 


8.6 


12.64 


20.56 


34.48 


53.82 


36.38 


84.04 


36.44 


187.035 


234.54 


13.79 


17.56 


10.23 


13.49 


23.09511111 


31.514 


33.77911111 


37.30044444 


33.41055556 


51.43011111 


63.45177778 


65.30333333 


62.49033333 


49.53544444 


90.668 


98.02044444 


120.0878889 


151.6648889 


133.5595556 


167.3923333 


164.6716667 


211.6618889 


218.6645556 


232.19 


397.21 


200.815 


397.34 


397.32 


397.25 


157.065 


374.82 


397.32 


397.29 


397.28 


397.21 


397.18 


397.21 


397.3 


397.27 


397.22 


397.22 


397.15 


397.18 


397.18 


347.76 


397.26 


228.5846667 


251.8702222 


210.685 


267.2154444 


248.1885556 


268.2436667 


199.856 


240.1583333 


248.7101111 


289.1445556 


269.571 


300.5576667 


305.3174444 


230.4615556 


267.1468889 


264.0756667 


267.5594444 


270.3647778 


289.0781111 


246.3966667 


257.4742222 


244.0626667 


286.964 


269.8262222 


291.7905556 


302.0175556 


275.9645556 


301.334 


314.7288889 


282.5782222 


276.6954444 


223.9163333 


284.5931111 


274.158 


278.7238889 


315.345 


289.3608889 


304.7337778 


283.4957778 


315.8652222 


343.1457778 


307.2334444 


306.6426667 


331.5072222 


319.8943333 


321.0216667 


9.3.4 Score attained in modified game settings when varying hidden layer 


neurons (Pipe Height 110) 


464 


1 32 16 (Anomaly) 47 


2 34 5 67 30 


3 0 


5 0 


21 


9.3.5 Score attained in modified game settings when varying hidden layer 


neurons (Pipe Height 105) 
1 2 2 1 4 


2 7 2 1 19 


9.3.6 Score attained in modified game settings when varying generations (2 


Hidden Layer Neurons) 
В eset 
150 4 
151 
152 
153 
154 
155 
156 


157 


158 


9.3.7 Score attained in modified game settings when varying generations (15 


Hidden Layer Neurons) 
150 4 4 2 7 


151 7 4 


152 4 4 


9.3.8 Score attained in modified game settings when varying generations (30 


Hidden Layer Neurons) 
150 24 


151 7 
152 12 
153 
154 
155 
156 
157 
158 
159 


160 


